The approaches described in this section could be pursued but are not necessarily approaches that have previously been conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.
Workloads for distributed computer networks typically include many separate tasks, also referred to as jobs. Each job may have a set of resource requirements for the job to be successfully executed. In order to efficiently dispatch a workload to a distributed compute farm, a coherent global view of both the workload and the current state of the compute nodes may be needed. A compute farm may include a collection of computer servers, usually maintained to accomplish server needs beyond the capability of a single machine. Having all the information available to a single entity allows that entity, sometimes referred to as a batch-processing master, to schedule jobs as resources free up. Such an approach ensures that a job is always scheduled as soon as the resources are available and policies permit.
Alternative systems are possible where multiple peer agents collectively determine when and where jobs can be submitted. In practice, such systems are difficult to manage due to the complexity and the lack of a single coherent view of the state of resources and jobs.
Conventionally, commercial products use a single job scheduler approach. However, the single job scheduler approach may face scalability problems. A single job scheduler needs to handle all job submissions, client queries, resource tracking, and job scheduling. The single job scheduler can be characterized by the number of jobs per second that the single job scheduler can handle. The number of jobs per second of the single job scheduler may depend on the number of entities (hosts or slaves) in the compute farm and the number of user queries concerning jobs and compute farm state. Eventually the single job scheduler may not be able to manage job scheduling, and resources may become idle for a longer time while jobs may be waiting to be dispatched. The remedy to such lack of scalability of the single job scheduler may include partitioning the compute farm into two or more segments, each with its own separate scheduler. However, such partitioning may inevitably lead to wasting resources since a job submitted to one segment of the compute farm cannot run on the other even if resources are available.